Combining lexical and statistical translation evidence for cross-language information retrieval

This paper explores how best to use lexical and statistical translation evidence together for CrossLanguage Information Retrieval (CLIR). Lexical translation evidence is assembled from Wikipedia and from a large machine readable dictionary, statistical translation evidence is drawn from parallel corpora, and evidence from co-occurrence in the document language provides a basis for limiting the adverse effect of translation ambiguity. Coverage statistics for NTCIR queries confirm that these resources have complementary strengths. Experiments with translation evidence from a small parallel corpus indicates that even rather rough estimates of translation probabilities can yield further improvements over a strong technique for translation weighting based on using Jensen-Shannon divergence as a term association measure. Finally, a novel approach to post-translation query expansion using a random walk over the Wikipedia concept link graph is shown to yield further improvements over alternative techniques for post-translation query expansion. Evaluation results on the NTCIR-5 English-Korean test collection show statistically significant improvements over strong baselines.